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DETAILED ACTION 



1 . This office action is responsive to the following communications: Application, 
filed on 09/21/2000; IDS, paper #2, filed on 09/21/2000; Request for reconsideration, 
filed 03/04/2003; Amendment A, filed 10/06/2003. 

2. Claims 1-30 are pending in this application. Claims 1, 3, 5, 7, 9, 10, 12, 13, 15, 
16, 18, 20, 22, 24, 25, 27, 28 and 30 are independent claims. 

3. The present title of this application is "Method for separating background sprite 
and foreground object and method for extracting segmentation mask and the apparatus" 
(as originally filed). 



4. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
12/01/2003 has been entered. 

Claim Rejections - 35 USC § 102 

5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 



Continued Examination Under 37 CFR 1.114 
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e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

6. Claims 1-15 are rejected under 35 U.S.C. 102(e) as being anticipated by Crinon 
et al., U.S. Patent Number 6,249,613, "Mosaic Generation and Sprite-Based Coding 
with Automatic Foreground and Background Separation", class 382/236. 

As per independent claim 1 , a foreground object and background sprite 
separation and extraction method for extracting . . . comprising the steps of: obtaining a 
global motion for transforming a coordinate system between a reference . . . ; mapping 
an original image corresponding to said frame into a reference coordinate system . . . ; 
generating a provisional sprite . . . ; cutting out a first image . . . using said global 
motion; obtaining a difference image between said first image and said original image; 
extracting a foreground object image . . . and extracting other region as a background 
image; mapping said background image . . . Crinon discloses segmenting foreground 
and background objects, coding in global motion parameters, col. 2, lines 37-55. Crinon 
further discloses reconstructing a sprite for isolating the object having the most motion 
in the video sequence, col. 3, lines 23-35. 

With respect to dependent claim 2, cutting out a second image from said 
background sprite . .. ; obtaining a difference image . . . ; extracting a foreground object 
image as a region in said difference . . . Crinon discloses in figure 8. 
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As per independent claims 3 and 5, they are rejected based upon similar rational 
as above independent claim 1 respectively. 

With respect to dependent claims 4 and 6, they are rejected based upon similar 
rational as above dependent claim 2. 

As per independent claim 7, a segmentation mask extraction method . . . 
comprising the steps of: receiving a foreground mask image . . . ; providing a first value 
as an alpha value ... in each of the first macro-blocks . . . ; providing said first value as 
said alpha value to all shape pixels in each of second macro-blocks . . . ; and outputting 
said segmentation mask. Crinon discloses first and second macro-blocks tagged as 
foreground and background objects, figure 9, col. 9, lines 42-67 and in col. 10, lines 1- 
30. 

With respect to dependent claim 8, receiving each of third macro-blocks ... as 
said background part; and providing said first value to said third macro-block . . . Crinon 
discloses a binary segmentation map and further using a neighborhood of macro-blocks 
around a macro-block of interest, col. 9, lines 20-28. 

As per independent claim 9, a segmentation mask extraction . . . comprising the 
steps of: receiving a foreground mask image; generating a number map . . . ; initializing 
a foreground map; providing a predetermined value to each of positions in said 
foreground map . . . ; providing said predetermined value to each of position in said 
foreground map . . . ; and generating said segmentation mask . . . Crinon discloses 
generating macro-blocks having multiple local motion type vectors and further having 
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macro-blocks as foreground only, background only, and foreground or background, col. 
2, lines 37-67. 

As per independent claims 10 and 13, they are rejected based upon similar 
rational as above independent claim 7. 

With respect to dependent claims 1 1 and 14, they are rejected based upon 
similar rational as above dependent claim 8. 

As per independent claim 12 and 15, they are rejected based upon similar 
rational as above independent claim 9, respectively. 

Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

8. Claims 16, 18, 20, 22-30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Wang, U.S. Patent Number 6,256,409, "Method for Determining a 
Correlation between Images using Multi-Element Image Descriptors", class 382/170. 

As per independent claim 16, a segmentation mask extraction method . . . 
comprising the steps of: obtaining said difference image by calculating an absolute 
difference . . . initializing an energy map for each macro-block of said difference image; 
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calculating energy values for said each macro-block; obtaining an average of said 
energy values; calculating a foreground ratio . . . ; and generating said segmentation 
mask . . . Wang discloses an energy map, feature vector or image descriptor to 
describe multi-band images or the correlation between a first image and a second 
image, col. 5, lines 12-59. However, it is noted that Wang fails to disclose representing 
an energy map as a macro-block for each different image. It would have been obvious 
to one of ordinary skill in the art at the time of the invention to include macro-blocks as 
image descriptors in the invention of Wang to further describe the correlation of the first 
and second images for performing image matching. 

As per independent claims 18 and 20, they are rejected based upon similar 
rational as above independent claim 16. 

As per independent claim 22, a segmentation mask extraction method for 
extracting a segmentation mask . . . comprising: a first step of regarding each of first 
macro-blocks as the foreground when an energy value of said first macro-block which is 
obtained . . . ; a second step of regarding each of second macro-blocks as the 
foreground . . . Wang discloses an energy map, feature vector or image descriptor to 
describe multi-band images or the correlation of a first image and a second image. 
However, it is noted that Wang fails to disclose representing an energy map as a 
macro-block for each different image. It would have been obvious to one of ordinary 
skill in the art at the time of the invention to include macro-blocks as the image 
descriptors disclosed in Wang to further describe the correlation of the first and second 
images for performing image matching. 
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With respect to dependent claim 23, iterating said second step for predetermined 
times. Wang further discloses re-processing and re-examining the image descriptors, 
col. 21, lines 4-17. 

As per independent claim 24, Wang further discloses comparing the image 
descriptors and categorizing the different image descriptors and belonging to a different 
image category type, col. 17, lines 32-67. However, it is noted that Wang fails to 
disclose representing an energy map as a macro-block for each different image. It 
would have been obvious to one of ordinary skill in the art at the time of the invention to 
include macro-blocks as the image descriptors disclosed in Wang to describe the 
correlation of the first and second images for performing image matching. 

As per independent claims 25 and 28, they are rejected based upon similar 
rational as above independent claim 22. 

With respect to dependent claims 26 and 29, they are rejected based upon 
similar rational as above dependent claim 23. 

As per independent claims 27 and 30, they are rejected based upon similar 
rational as above independent claim 24 respectively. 

Allowable Subject Matter 

9. Claims 17, 19 and 21 are allowed. 

10. The following is a statement of reasons for the indication of allowable subject 
matter: The prior art cited in its entirety fail to render obvious dividing an energy value 
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by an average to obtain the macro-blocks to represent the energy values in the 
foreground and background maps. 

Response to Arguments 

1 1 . Applicant's arguments filed 03/04/2003 have been fully considered but they are 
not persuasive. 

In response to applicant's argument that the references fail to show certain 
features of applicant's invention, it is noted that the features upon which applicant relies 
(i.e., generating a provisional sprite one and using a provisional sprite to provide a clear 
sprite) are not recited in the rejected claim(s). Although the claims are interpreted in 
light of the specification, limitations from the specification are not read into the claims. 
See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

Applicant argues that Crinon fails to disclose the use of a provisional sprite. 
Crinon discloses the use of multiple sprites, col. 2, lines 37-55, to create a background 
sprite. Deleting foreground objects generates applicant's provisional sprite as claimed. 
Crinon discloses segmenting foreground and background objects. Applicant argues 
that Crinon fails to disclose separating the foreground and background image using the 
provisional sprite and discloses only automatic segmentation does not require any 
additional frame storage and works. The text recites that multiple sprites are used as 
needed and can include the use of two background sprites, col. 2, lines 37-40. 
Furthermore it is inherent that if the sprite is constructed for only one object in the image 
and the sprite becomes more useable since it is related to the one object that the 
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background image may be composed of multiple objects and therefore would require 
multiple sprites. 

Applicant argues that Crinon fails to disclose cutting out a second image form the 
background sprite by using global motion, obtaining a difference image . . . and 
extracting a foreground object image as a region in the difference image . . . higher than 
a threshold. Applicant states that in figure 8, Crinon indicates how the encoder 
distinguishes background from foreground, which Examiner interprets as obtaining a 
difference between images. Applicant further states that Crinon discloses global motion 
estimation residual are compared and if the global residual is greater than tagging the 
image as a foreground, which Examiner interprets as extracting a foreground object 
image where the difference value in the region is equal or higher than a threshold. 

Applicant argues that Crinon fails to disclose first and second macro-blocks 
tagged as and foreground and background objects and extracting a segmentation mask 
and a two-stage macro-block approximation. Crinon indicates how the encoder 
distinguishes background from foreground, which Examiner interprets as obtaining a 
difference between images. Applicant states that Crinon refers to a process 
segmentation map, making regions homogeneous and updating the mosaic according 
to a new segmentation map, which Examiner interprets as multi process segmentation 
procedure. 

Applicant argues that Crinon fails to disclose providing a value to a position in the 
foreground correspond to first macro blocks . . . value to each of the positions in the 
foreground map corresponding to second macro-blocks when the value of the number 
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map correspond to the second macro-block is equal to or larger than a second 
predetermined value. Applicant states that Crinon refers to generating macro-blocks 
having multiple local motion type vectors and macro-blocks as foreground only, which 
Examiner interprets as a macro-block equal to a first predetermined value, and 
foreground or background, which Examiner interprets as a macro-block equal to a 
second predetermined value. 

Applicant argues that Wang fails to disclose generating of a segmentation mask 
using foreground ratio. Wang discloses determining a correlation between images, col. 
2, lines 10-13, the use of an energy matrix referred to as an orientation map, calculating 
the dominant orientation at each pixel, and combining the descriptors to form an image 
descriptor, col. 5, lines 12-23. Therefore making it obvious to use the image descriptors 
disclosed in Wang with the mosaic images built from images of a scene and image 
segmentation, which is a correlation between images, as disclosed in Crinon. Applicant 
further argues that Wang does not describe first and second macro-block 
approximations. Crinon discloses sub-classification of the regions tagged as 1 , the 
foreground region, into 1a and 1b, for macro block coding, col. 6, lines 26-37. Crinon 
further discloses the tagged regions are coded for different image support. Wang 
discloses determining a correlation between images using an energy map. It is obvious 
that if Crinon allows for the sub-classification of the macro-blocks in the foreground 
region, the invention of Wang would allow for the coding of the sub-classification of the 
macro-blocks using energy map descriptors. 
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In response to applicant's argument that the examiner's conclusion of 
obviousness is based upon improper hindsight reasoning, it must be recognized that 
any judgment on obviousness is in a sense necessarily a reconstruction based upon 
hindsight reasoning. But so long as it takes into account only knowledge which was 
within the level of ordinary skill at the time the claimed invention was made, and does 
not include knowledge gleaned only from the applicant's disclosure, such a 
reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 
1971). 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Motilewa A. Good-Johnson whose telephone number is 
(703) 305-3939. The examiner can normally be reached on Monday - Friday 8:30 AM - 
5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers 
for the organization where this application or proceeding is assigned are (703) 872-9314 
for regular communications and (703) 872-9314 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 306- 
0377. 
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